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This paper describes an example of a course evaluation where the evaluation process becomes an important 
part of the learning process. The setting is an action-learning based course in an executive program. The 
participants apply a framework for perceiving processes to their own learning process . The framework is 
presented and experiences from the perspectives of the participants and the instructors are described and 
discussed. Tentative conclusions are drawn from the experiences of using this type of course evaluation , 
including the following examples: Make a clear distinction between the learning processes of the 
instructors and the ones of the participants; Put the responsibility for the evaluation of the learning process 
with the individual; Design the evaluation process in concert with the course contents . Finally , there is a 
general discussion on how evaluation can be used as a lever of learning. 



INTRODUCTION AND BACKGROUND 

Often the evaluation process of a course is 
considered as a process following the actual 
learning process. In this paper an example of a 
course evaluation is described and discussed 
where the evaluation process is an important part 
of the learning process in itself. Having 
experiences from traditional forms of course 
evaluation where the participants are to fill out 
forms to rate different aspects of the course, we 
(i.e. the instructors of the course) wanted to move 
to something else. Our experiences were that this 
traditional kind of course evaluation seldom are 
useful for the instructors, and even less useful for 
the participants. They are usually kindly asked to 
spend a few minutes of their time to fill out some 
forms in order to help the instructors to improve 
the course for future participants. Some people 
tend to spend much effort trying to answer the 
different questions, while other very quickly 
answer the different questions: if they "like" the 
course in general terms they give a high grade in 
all different categories, or if they did not like the 
course, vice versa. 

With a traditional approach we as instructors ask 
the participants for help in our learning 
processes. We want to learn more about the 
course and about how the next version of it could 
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be improved. One problem though, is that a 
traditional evaluation form (with graded scales 
like 1 to 5, A to E, or similar) is not very helpful 
for our learning. How should I improve the course 
and/or my performance based on the results from 
the evaluation? The figures do give important 
information regarding how the participants’ 
perceived the course, the performance of the 
instructors, and to some extent how much they 
learnt and/or how useful they perceived the 
course and its contents. In our experiences, these 
results from course evaluations are not always 
very helpful for our learning, for the participants' 
learning, or for the improvement of the course. 
The outcome may include ideas of areas where 
there are particular needs for improvements, but 
what do we know about the quality of the results, 
about the actual learning of the participants? 



We decided to start using another type of course 
evaluation, where we moved away from this 
traditional form of course evaluation. (Worth 
.noting is that there are of course different types 
of course evaluations. For the purpose of this 
discussion, I talk about "traditional” course 
evaluation, which means a number of questions 
there the participants can rank the course and 
the instructors on a graded scale.) We aimed at a 
course evaluation process where both the 
participants and we as instructors could learn 
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something. We wanted to offer the participants 
something helpful and useful for their own 
learning processes, and not only ask them to help 
us to help future participants. 

The result of our efforts was to make use of one of 
the models for perceiving change processes that 
the participants had used during the course. 
Before discussing the setting for this work and 
the findings from this evaluation process, the 
next section will describe some theoretical links 
for our attempts to find forms for evaluation as 
part of the learning processes of the participants. 

THEORETICAL LINKS 

A Framework for Perceiving Processes 

The course evaluation presented here is an 
attempt to link the evaluation process to the 
learning processes of the participants. An 
important additional aim is to produce valuable 
input for the instructors' learning processes. The 
form, as described above, is to let the participants 
reflect at the end of the program. The use of the 
framework in this reflection enables the 
participants to reflect on two different levels: on 
both a task and a person level. In the program 
several models for handling change processes are 
used, most of them described in Lundeberg 
(1993). The model used for the evaluation is a 
framework for perceiving processes for the input 
and output of a process and is called the "X- 
model" (Lundeberg, 1993, section 1.22), see figure 
1 below. 

The essence of the X-model is: Each process in a 
business firm includes personal behavior and 
task processes in an inseparable whole. The main 
idea behind the framework is to take both 
personal behavior oriented and task oriented 
matters into consideration in a change process. 



FIGURE 1 



THE X-MODEL 




Action and Reflection 

The importance of combining action and 
reflection (see e.g. Schon,1983) is a crucial point 
in the program where we introduced the course 
evaluation. Kolb (1984) discusses this 
combination in terms of "two dialectically 
opposed modes of transforming experience, one 
via intentional reflection, the other via 
extensional action". In order to include both 
components of action and reflection in our 
program, the participants work in cross- 
functional teams in customer projects to 
accomplish action. For the reflection, the 
participants use different types of reflection 
instruments during the program and especially 
the evaluation process at the end of the program, 
focussed in this paper. 

The combination of action and reflection is one 
important dimension, another is the combination 
of theory and practice (see e.g. Argyris and Schon, 
1974). In our program the practice-part is the 
work with customer cases. The theory used is 
presented at seminars and applied in the 
customer cases. The reflection will then help the 
participants to learn from the theory and practice 
combined. There has to be a willingness to learn 
though. Schein (1992) describes this like: 
"Learning and change cannot be imposed on 
people. Their involvement and participation are 
needed diagnosing what is going on, figuring out 
what to do, and actually doing it." 
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One purpose with the evaluation described in this 
paper was to find a way for the participants to 
reflect on their actions and learn. Then this in 
turn would produce valuable input for the 
improvement of the program. The difficulties for 
managers to reflect is illustrated by Senge (1990). 
He describes experiments which have shown that 
even if there is time for reflection most managers 
do not reflect carefully on their actions, but think 
about strategies for their next action. Here, we 
tried to build in reflection in an "action item", i.e. 
the "reflection assignment" every participant has 
to do before the concluding meeting. 

At many educational institutions there is a move 
from traditional teaching towards new forms of 
teaching in order to improve the learning. Cowen 
(1996) describes some aspects of this change 
process at one major business school. When 
discussing a learning perspective he says: "the 
most important point about a true learning 
perspective is that is may require a wholly 
different approach to how business schools teach 
and organise themselves". When trying to break 
old structures at Weatherhead School of 
Management at Case Western Reserve 
University, they, among many other things, 
introduced new student assessment 
arrangements and learning plans (Cowen, 1996). 

In the next section, the setting for our new course 
evaluation is described. First, there is a short 
general description of the course, then the section 
includes a description of the evaluation of the 
course evaluation. 

THE SETTING 

The "Course" 

The course is an action-learning based 
competence development program within the 
area of Information Management in an executive 
program. The focus of the program is on handling 
change processes, and in this particular setting, 
change processes in the sales-relation with 
customers. The participants are from the 
telecommunication industry and each program 
has 15-16 participants forming three to four 
customer teams, with one account manager 
responsible for each team. The other team 
members come from different units within the 
organization. By working in cross-functional 
teams during the program, the degree of mutual 



understanding and sense of a common mission is 
strengthened. 

Some key issues of the program are: 

♦ It is an "in-house" program within one 
organization. 

♦ Executives are involved in the program and 
participate when the projects are discussed. 

♦ There is a "real-life" context, i.e. the 
participants are to achieve business results in 
their customer cases. 

♦ Projects in cross-functional teams are in focus. 

♦ Methods and models are included as support for 
the teams' work with their projects. 

The teams work on projects together with their 
customers and their mission is to achieve 
business results from the projects during the 
program, which runs over a time period of about 
three months. The cross-functional projects are in 
focus in the course, and the project work is mixed 
with theoretical seminars aimed at supporting 
their work in the projects. 

The first step is that every participant produces a 
personal development plan which is discussed 
with her/his superior. Then there is a 2V2-day 
kick-off for the program where the working 
methods are introduced and applied to prior 
customer cases (including video-taped sessions). 
The program then consists of three full-day 
meetings with one month's interval, where the 
customer cases are reviewed and discussed. The 
executives are present at these meetings. Each 
team has prepared and distributed a written 
project- report to all other participants in advance 
of each meeting. Between these full-day meetings 
there are half-day meetings held where methods 
and experiences of the participants are discussed. 
These seminars may include theoretical moments 
judged to be important to support the customer 
cases. 



The design of the program is briefly described in 
table 1. 
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TABLE 1 



THE DESIGN OF THE PROGRAM 



TIME 


ACTIVITY 




A personal development plan is produced by 
each participant in cooperation with her/his 
superior 


m days 


Kick-off for the program 


1 day 


Process-meeting 1 : 

All customer cases are discussed 


Vi day 


Seminar for theoretical presentations and 
exchange of experiences: Models and 
frameworks are presented, applied and 
discussed with links to the customer cases 


1 day 


Process-meeting 2: 

All customer cases are discussed 


Vi day 


Seminar for theoretical presentations and 
exchange of experiences: Models and 
frameworks are presented, applied and 
discussed with links to the customer cases 


1 day 


Process-meeting 3: 

All customer cases are discussed 


Vz day 


Follow-up discussion with superior 
Concluding meeting: The final reflective 
assignment ("course evaluation") is discussed 
and linked to the results in the customer cases 



At the end of the program each participant meets 
her/his superior to follow-up on the personal 
development plan. Then before the final 
concluding meeting each participant is asked to 
apply a framework for perceiving change 
processes (the X-model) on their own learning 
process during the program. These descriptions of 
the learning processes of the participants are 
then summarized (anonymously) and discussed 
at the last session. 

Finally, a short note on why this section was 
named the "course" within quotes. We did not call 
it a course, but a "competence development 
program". The word "course", we found, to some 
people meant going away from work, as a sort of 
break. We wanted to have close links to the work 



with the customers and participants who felt that 
the time spent at the program was their work, 
although with a special learning focus. 

Using the X-Model for Evaluation 

At the end of a program we ask every participant 
to apply the X-model, which they by this time are 
familiar with from the program, on their own 
learning process during the program. This means 
that the focus of the evaluation process is put on 
their own learning processes. They have to build 
the description of their learning process with the 
help of the model and are not given any specific 
questions to answer. Their descriptions include 
their expectations, the process during the course, 
the lessons they have learnt from the course. 

Before the concluding meeting every participant 
send their "X-model" to us and we combine all X- 
models from the participants into one document. 
In the process to combine the models we remove 
all names to make it anonymous. (One reason for 
this is to reduce the risk that their X-models 
would be restricted by the fact that their 
superiors read the summary of all X-models.) The 
document based on all the participants' X-models 
is then distributed and discussed at the 
concluding meeting. This means that there is a 
discussion at the end of the program where the 
learning experience during the program is 
discussed and different participants can compare 
their own learning experience with other persons' 
experiences. 

Evaluation of the Evaluation 

Our own experience, is that this form of 
evaluation has worked very well. The 
participants spend time and efforts in the 
reflective assignment to apply the X-model to 
their participation in the program. Some of them 
have made comments on the usefulness of 
actually sitting down, reflecting on the learning 
experience. 

In order to get a better picture than just our 
general impressions about the usefulness, we 
wanted to receive more feedback from the 
participants about their opinions about the 
evaluation. Therefore, we conducted telephone 
interviews with the participants about two 
months after the program. 13 out of the 15 
participants of the program were interviewed. 
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(One was on a leave and the other person had 
moved.) By giving them time before asking 
questions about the evaluation process we 
reduced the risk to mix up their own learning 
process on one hand, and the course process on 
the other hand. The interviews included 
questions both about the program in general, 
questions about the role of the evaluation, and 
their perception of the links to their learning own 
processes. 

By asking about the program, we had the 
opportunity to compare the findings from the 
interviews by the outcome from the X-models. 
That is, would the interviews after the completion 
of the program add much know knowledge or not? 

FINDINGS 

Participants’ Perspective 

The participants describe their participation in 
the program using the framework described 
above (the X-model). This means that they reflect 
on preconditions and expectations both regarding 
their own situation (person-level) and regarding 
the customer case their team worked with (task- 
level). They also reflect on the process, that is, 
both the process at the meetings and the process 
during the work in the team. Finally, the 
participants reflect on the outcome in terms of 
their own learning (person-level) and business 
results from the customer case (task-level). 

On the personal-oriented level, participants often 
emphasize development of personal skills which 
improve their handling of different customer 
situations and of developing relationships and 
business with customers. Concerning the 
customer cases they often mention to what extent 
these have been brought forward during the 
program. Below, a few examples of comments 
from the X-models are presented. 

Preconditions 

"I had high expectations on the program where I 
expected to learn new methods for working with 
customer projects and at the same time meet 
colleagues from different teams and exchange 
experiences." (person- level) 



"I saw the program as a challenge and as a step 
in my personal development as a salesman." 
(person-level) 

"The expectations of the program were very 
blurred." (person-level) 

"To work with real' cases was great, and to me 
our customer case was very exciting, as I have 
other similar cases." (task-level) 

Process 

"The process-meetings improved over time. We 
went from organized chaos to organized meetings 
with focus on specific questions." 

"The design of the process-meetings with 
observers led to valuable comments after the 
meetings." 

"The strength of the program is the common 
group where similar customer cases can be 
discussed with persons with different knowledge 
and experiences." 

Outcomes: 

"The importance to stop for a while and think 
about purpose and objective, both for a specific 
meeting and for a project in general." (person- 
level) 

"I thought that the customer cases should be in 
another phase - the reality was not like that - 
processes in business cases in our organization 
take time!" (task-level) 

"A structured approach to customer-related 
work." (task-level) 

"An increased understanding of the process 
approach." (person-level) 

Before the concluding working-meeting we (i.e. 
the instructors) combine the X-models into one 
document (as described above). This means a 
document consisting of about 10-12 pages with 
comments sorted in person- and task-levels 
regarding preconditions, process, and outcome. 
When this document is discussed at the 
concluding meeting, both personal learning and 
business results in the customer cases are 
included in the discussion. At the concluding 
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meeting there are possibilities for the 
participants to compare their reflections, and to 
discuss how to continue the learning process after 
the program. What then do we learn from this as 
instructors? This will be discussed below. 

Instructors’ Perspective 

The document describing all participants' 
reflection of the program gives a very rich picture 
of several different dimensions. We have found 
that this form of evaluation links much more 
directly to the learning processes of the 
participants then traditional forms of evaluation. 
Instead of comments on how the participants 
have perceived the program we get their own 
reflections of their own learning. This means that 
they focus on their own learning. If participants 
are asked to evaluate a course or a program they 
are asked to change the focus from their own 
learning process to the course-development 
process. 

From their reflective comments we learn about 
what they have learnt - and about what they have 
not learnt (or at least not mention in their X- 
models). We can see how different messages have 
been interpreted by the participants, and in some 
cases also applied in the customer cases. When 
trying to improve the program, this kind of 
comments have proved to be very useful. One 
example is when people made comments about 
"blurred expectations" we then tried to improve 
and clarify the information given to the 
participants before the program starts. The rich 
picture built from all comments about the 
learning experiences of the participants, helps us 
to identify what parts of the program the 
participants really have learnt and used - and 
vice versa. 

The discussion at the concluding meeting is a 
very useful supplement to the written reflections. 
Here, we can follow a discussion where the 
participants (and the executives involved in the 
program) discuss the entire program. The 
discussion is not in terms of evaluate the 
program, but to focus on what different people 
have learnt, how the gained knowledge can be 
applied in the future, how and to what extent the 
process has been useful for the customer case and 
for the personal learning. 



Results of the Evaluation 

The interview survey of the participants 
confirmed our picture about the usefulness of this 
form of evaluation. The participants usually had 
no changes in their opinions about the program. 
They regarded this final reflection assignment, 
when using the X-model for their participation, as 
useful for themselves. Some participants 
mentioned that they had spent a considerable 
amount of time on the reflection assignment, as 
they had found it interesting to thoroughly walk 
through the whole program and think about 
everything that had happened during the 
program. 

In general, the opinions about the program were 
consistent over time, i.e. there were very few 
differences between the opinions at the end of the 
program and those two months later. There was 
one exception though, where there was a shift in 
opinion between the end of the program and the 
interviews. At the end of the program there were 
some comments about the need to reduce the 
number of reflection instruments during the 
program. Some participants expressed that it was 
too many reflection instruments to fill out at 
every meeting. After about two months when the 
interviews were conducted, some of these 
comments had changed. One person expressed 
this like the following: 

"Well, I remember that I complained 
about the number of reflection 
instruments, but now when I look back at 
the process I realize the usefulness of 
them. I must say that after all I think you 
should stick to the number of reflection 
instruments next year, but be aware that 
they will complain too!" 

This comment could be seen as an example of how 
the usefulness of reflection sometimes is not seen 
immediately. In the next section, this example 
and other aspects of the findings will be 
discussed. 

DISCUSSION 

After having used this form of course evaluation, 
our general impression is that both the 
participants and we as instructors have gained 
from it. This evaluation is an important part of 
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the learning processes of the participants, as well 
as important input for us as instructors when 
redesigning the program. 

The evaluation used for the program is in line 
with the program in at least two different ways: 
we use a framework which the participants are 
familiar with (the X-model), and we use an 
individual reflection assignment as our 
evaluation of the program, which is in line with 
the reflection instruments used throughout the 
program. 

In our opinion, there are two main objectives for 
this type of course evaluation where the focus is 
on the learning processes of the participants. The 
first one is to link the evaluation closer to the 
learning processes of the participants. The second 
objective is to separate the learning processes of 
the participants from those of the instructors. 

There are several risks with traditional forms of 
course evaluation when participants fill out forms 
with different types of questions about the course. 
There is the risk of confusing the learning 
processes of the participants with the ones of the 
instructors. Asking the participants to give 
feedback on the course while they are in their 
own learning process, is to ask for something 
difficult. The risk is that the feedback neither is 
useful for the instructors when improving the 
course, nor facilitate the participants’ learning. 

The participants are, of course, very important 
sources of information when trying to improve a 
course. One challenge for instructors is to find 
ways to receive this feedback with as good quality 
as possible, i.e. to get feedback which is useful in 
the work to redesign a course. We have found that 
the approach "evaluation as learning" results in 
useful feedback for us as instructors and at the 
same time the evaluation is useful for the 
participants as it helps them reflect on their own 
learning. This is a help that might be important 
in accordance with the discussion by Senge (1990) 
about the difficulties for managers to take the 
time for reflection. Here they are "forced" to 
reflect on their participation of the program and 
often the participants find it useful, at least 
afterwards, as the example from the interview 
above illustrates. 



One key factor is, in our opinion, that we have put 
the focus on the participants’ learning. Ironically, 
we have found that by doing so, we learn more as 
instructors as well! 

The description shows that we have found this 
form of course evaluation very useful. What are 
the drawbacks? 

This qualitative course evaluation may cause 
problems for those looking for measures which 
are quantitative and possible to easily compare 
between different courses and different years. If 
one looks for that kind of measures it is important 
to ask the question: what is the rationale of this 
need? We have focussed on the learning of the 
participants as the most important, and as a 
second important component how we as 
instructors can learn and improve the program. 

Is this form of evaluation possible to use in other 
courses? My spontaneous answer is simply: yes. 
There are some special preconditions for our use 
though, which means that the evaluation still has 
to be tried out in a different context. In our 
program the participants are familiar with the 
framework (the X-model) from their work during 
the program. If this particular framework would 
be used in another context there is a need for a 
short introduction to it. As the model is fairly 
simple and intuitive, this should be possible to do 
if even the course dealt with something totally 
different than change processes. This could mean 
1-2 pages of instructions and a short presentation 
of the model when the "reflection assignment" is 
introduced. In some senses all courses deal with 
the same thing: to help participants to gain new 
knowledge during a certain period of time - to 
learn. 

The evaluation described here could, of course, be 
improved. One idea is to stretch it over time. For 
example, the participants could be given a 
reflection assignment during the program, and 
maybe also a number of months after the 
program has finished. This could help stretching 
the learning process over a longer period of time. 
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CONCLUDING REMARKS 
AND SOME LESSONS 

In this paper on evaluation as learning, one 
purpose has been to illustrate a form of course 
evaluation which we have used, and found being 
helpful both for the participants of the program, 
as well as for us as instructors. One key element 
is that the evaluation rather is a form for the 
participants to reflect on their own learning 
during the program, than an evaluation of a 
course. When the participants apply the 
framework (the X-model described above), this 
means that the focus is entirely on their own 
learning processes. They perceive the course 
evaluation, or rather their reflection assignment, 
as useful for their own learning. As a nice side- 
effect, it is most useful as a means for improving 
the future programs. 

From the use of this evaluation, where we have 
tried to link evaluation closely to learning, there 
are several lessons to be drawn. Below some 
tentative conclusions are presented: 

Lesson 1: Make a clear distinction between the 
learning processes of the instructors and the ones 
of the participants. 

One purpose of course evaluation is to receive 
feedback in order to improve the course. This 
means that the focus is on the learning process of 
the instructors. When participants are asked to 
give feedback on the course while they are in the 
middle of their own learning process is to ask 
them for some thing difficult. We have found that 
we as instructors learn more by focussing on the 
learning processes of the participants. The input 
from the participants is, in our opinion, of high 
quality when the participants who spend time 
and efforts doing the evaluation (or reflection 
assignment) perceive it as useful in their own 
learning process - they gain something from it 
themselves. 

Lesson 2: Put the responsibility for the evaluation 
of the learning process with the individual. 

Evaluation of the learning process is something 
personal which by definition must be carried out 
by the person herself. By giving the responsibility 
to the course participants, without any detailed 
guidelines as specific questions, they have to 
reflect on their own learning process and involve 



themselves in the process. They have to involve 
themselves more than what is necessary when 
answering some questions on a graded scale. 

Lesson 3: Design the evaluation process in 
concert with the course contents. 

In this case we have used a framework included 
in the program and let the participants apply it to 
their own learning processes. When applying 
something from the course contents in the 
evaluation process, the learning can be improved 
and the risk of mixing the learning processes of 
the participants and of the instructors is reduced, 
and at the same time it is an example of "walk the 
talk". The possibilities for doing this varies of 
course between different courses. If there are no 
natural links to reflection and evaluation from 
any framework used in the course, a general 
model (like the model we have used) could be 
applied. The difference is that there is a need for 
a short introduction to the model before the 
reflection assignment. 

Lesson 4: Clarify the underlying purposes of the 
evaluation. 

An important question to ask is: for whom is the 
evaluation done? Is it primarily for the 
instructors? For the participants? For both? For 
future participants? For someone else? 
Traditionally, course evaluations often are for the 
instructors to improve the course, which in turn 
future participants will gain from. In our 
example, the evaluation is done both for the 
participants and for the instructors (in that 
order). We wanted to help them to further their 
learning, and at the same time we could learn in 
order to improve the program. When designing a 
course evaluation suitable for the situation it is 
crucial to clarify the underlying purposes: for 
whom and why are we doing this? 

Lesson 5: Use the evaluation for reflection to 
improve learning. 

We have designed this evaluation as a "reflection 
assignment". Instead of just asking the 
participants to fill out an evaluation, we have 
used the evaluation as a tool for their reflection in 
order to improve their own learning. By 
refr ami ng "evaluation" the time and efforts spent 
on it could be changed, and by this also the 
usefulness of it. 
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In this paper, an example of a course evaluation 
has been presented and some experiences from 
this. There are obviously much more to be done in 
this area. This is simply an example of some steps 
in a direction where evaluation is more closely 
linked to learning processes - evaluation as 
learning. 

ENDNOTE 

The evaluation of this course evaluation was 
conducted as a teaching project within the 
International Teachers Program (ITP) 1996/97 at 
London Business School (see Robertson and 
Morrison, 1996). 
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